Encoding and decoding a stylized custom graphic

ABSTRACT

Disclosed are methods for encoding information in a graphic image. The information may be encoded so as to have a visual appearance that adopts a particular style, so that the encoded information is visually pleasing in the environment in which it is displayed. An encoder and decoder are trained during an integrated training process, where the encoder is tuned to minimize a loss when its encoded images are decoded. Similarly, the decoder is also trained to minimize loss when decoding the encoded images. Both the encoder and decoder may utilize a convolutional neural network in some aspects to analyze data and/or images. Once data is encoded, a style from a sample image is transferred to the encoded data. When decoding, the decoder may largely ignore the style aspects of the encoded data and decode based on a content portion of the data.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.17/302,361, filed on Apr. 30, 2021, which is a continuation of U.S.patent application Ser. No. 16/846,949, filed on Apr. 13, 2020, andissued as U.S. Pat. No. 11,024,058, which is a continuation of U.S.patent application Ser. No. 16/022,536, filed on Jun. 28, 2018, andissued as U.S. Pat. No. 10,657,676, each of which are incorporatedherein by reference in their entireties.

TECHNICAL FIELD

The present disclosure generally relates to the technical field ofencoding and decoding. Specifically, embodiments encode and decode bitstrings using machine learning algorithms.

BACKGROUND

Uses for social networks are expanding. Originally conceived as a methodto make it easier to keep in touch with friends, social networks are nowproviding front end functionality for a variety of Internetcapabilities, including news, ecommerce, messaging, classifiedadvertisement, and other features. To that end, the easier it is forusers to employ the features of social networks to solve problemsspecific to their needs, the more readily the users adopt thesefeatures. Therefore, improvements in the ease of use of socialnetworking features are desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings.

FIG. 1 is an exemplary block diagram of a messaging system forexchanging data (e.g., messages and associated content) over a network.

FIG. 2 is block diagram illustrating further details regarding themessaging system, according to some embodiments.

FIG. 3 is a schematic diagram illustrating data which may be stored inthe database of the messaging server system, according to certainexample embodiments.

FIG. 4 is a block diagram illustrating functional components of a customencoding system.

FIG. 5 is an example of a graphic according to at least some of thedisclosed embodiments.

FIG. 6 is a flowchart of a method for training an encoder and a decoder.

FIG. 7 is a flowchart of a method of encoding a bit string.

FIG. 8 is a flowchart of a method of decoding a bit string from imagedata.

FIG. 9 is a block diagram illustrating an example software architecture,which may be used in conjunction with various hardware architecturesherein described.

FIG. 10 is a block diagram illustrating exemplary components of amachine

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments of the disclosure. In the followingdescription, for the purposes of explanation, numerous specific detailsare set forth in order to provide an understanding of variousembodiments of the inventive subject matter. It will be evident,however, to those skilled in the art, that embodiments of the inventivesubject matter may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures, andtechniques are not necessarily shown in detail.

The disclosed embodiments may provide for encoding of a user defined bitstring into a visual image. The visual image may then be distributed bythe user. Later, other users, or the creating user in some aspects, mayscan the visual image to identify the bit string. Previous solutions tothis problem presented an unappealing visual appearance. For example,Q/R codes, while functional, may visually subtract from an environmentin which they appear. Thus, it may be desirable to create a visualappealing way of encoding information that may more readily be adoptedinto a visual environment, without unduly burdening the visualenvironment with technical appearing designs.

To accomplish this, an encoder and decoder are trained as a pair toencode and decode bit strings. The training is accomplished by tuningthe encoder and decoder to minimize a loss occurring during the combinedencoding and decoding process. After the encoder encodes a bit string,the resulting image data is made more visually appealing by transferringstyle information from a sample style image to the image data encodingthe bit string. The visually appealing image data is then placed withina graphic. When a user attempts to decode the graphic, first, an imagecontaining the graphic is scanned. The graphic may then be detected inthe scanned image. Image data is then extracted from the graphic, anddecoded by the decoder to identify the encoded bit string. The encodedbit string may represent one or more operations to be performed by asocial network after its decoding. In some aspects, the encoded bitstring encodes actual instructions for the social network, such as thoseincluded in an intermediate language like Common Intermediate Language(CIL) or in a scripting language such as Perl or Python. In some otherembodiments, a social network may maintain a database, or other stablestorage to store a mapping from bit string information to one or moreoperations. When a bit string is encoded, a mapping entry for the bitstring may be added to the table. The table may identify operations toperform when the bit string is decoded.

FIG. 1 is a block diagram showing an example messaging system 100 forexchanging data (e.g., messages and associated content) over a network.The messaging system 100 includes multiple client devices 102, each ofwhich hosts a number of applications including a messaging clientapplication 104. Each messaging client application 104 iscommunicatively coupled to other instances of the messaging clientapplication 104 and a messaging server system 108 via a network 106(e.g., the Internet). As used herein, the term “client device” may referto any machine that interfaces with a communications network (such asthe network 106) to obtain resources from one or more server systems orother client devices. A client device may be, but is not limited to, amobile phone, desktop computer, laptop, portable digital assistant(PDA), smart phone, tablet, ultra book, netbook, laptop, multi-processorsystem, microprocessor-based or programmable consumer electronicssystem, game console, set-top box, or any other communication devicethat a user may use to access a network.

In the example shown in FIG. 1 , each messaging client application 104is able to communicate and exchange data with another messaging clientapplication 104 and with the messaging server system 108 via the network106. The data exchanged between the messaging client applications 104,and between a messaging client application 104 and the messaging serversystem 108, includes functions (e.g., commands to invoke functions) aswell as payload data (e.g., text, audio, video, or other multimediadata).

The network 106 may include, or operate in conjunction with, an ad hocnetwork, an intranet, an extranet, a virtual private network (VPN), alocal area network (LAN), a wireless LAN (WLAN), a wide area network(WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), theInternet, a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a plain old telephone service (POTS) network,a cellular telephone network, a wireless network, a Wi-Fi® network,another type of network, or a combination of two or more such networks.For example, the network 106 or a portion of the network 106 may includea wireless or cellular network and the connection to the network 106 maybe a Code Division Multiple Access (CDMA) connection, a Global Systemfor Mobile communications (GSM) connection, or another type of cellularor wireless coupling. In this example, the coupling may implement any ofa variety of types of data transfer technology, such as Single CarrierRadio Transmission Technology (1×RTT), Evolution-Data Optimized (EVDO)technology, General Packet Radio Service (GPRS) technology, EnhancedData rates for GSM Evolution (EDGE) technology, third-GenerationPartnership Project (3GPP) including 3G, fourth-generation wireless (4G)networks, Universal Mobile Telecommunications System (UMTS), High-SpeedPacket Access (HSPA), Worldwide Interoperability for Microwave Access(WiMAX), Long-Term Evolution (LTE) standard, or others defined byvarious standard-setting organizations, other long-range protocols, orother data transfer technology.

The messaging server system 108 provides server-side functionality viathe network 106 to a particular messaging client application 104. Whilecertain functions of the messaging system 100 are described herein asbeing performed by either a messaging client application 104 or by themessaging server system 108, it will be appreciated that the location ofcertain functionality either within the messaging client application 104or the messaging server system 108 is a design choice. For example, itmay be technically preferable to initially deploy certain technology andfunctionality within the messaging server system 108, but to latermigrate this technology and functionality to the messaging clientapplication 104 where a client device 102 has a sufficient processingcapacity.

The messaging server system 108 supports various services and operationsthat are provided to the messaging client application 104. Suchoperations include transmitting data to, receiving data from, andprocessing data generated by the messaging client application 104. Thisdata may include message content, client device information, geolocationinformation, media annotation and overlays, message content persistenceconditions, social network information, and live event information, asexamples. Data exchanges within the messaging system 100 are invoked andcontrolled through functions available via user interfaces (UIs) of themessaging client application 104.

Turning now specifically to the messaging server system 108, anApplication Programming Interface (API) server 110 is coupled to, andprovides a programmatic interface to, an application server 112. Theapplication server 112 is communicatively coupled to a database server118, which facilitates access to a database 120 in which is stored dataassociated with messages processed by the application server 112.

The API server 110 receives and transmits message data (e.g., commandsand message payloads) between the client device 102 and the applicationserver 112. Specifically, the API server 110 provides a set ofinterfaces (e.g., routines and protocols) that can be called or queriedby the messaging client application 104 in order to invoke functionalityof the application server 112. The API server 110 exposes variousfunctions supported by the application server 112, including accountregistration; login functionality; the sending of messages, via theapplication server 112, from a particular messaging client application104 to another messaging client application 104; the sending of mediafiles (e.g., images or video) from a messaging client application 104 tothe application server 112, for possible access by another messagingclient application 104; the setting of a collection of media data (e.g.,story); the retrieval of a list of friends of a user of a client device102; the retrieval of such collections; the retrieval of messages andcontent; the adding and deletion of friends to and from a social graph;the location of friends within a social graph; and the detecting of anapplication event (e.g., relating to the messaging client application104).

The application server 112 hosts a number of applications andsubsystems, including a messaging server application 114 and a socialnetwork system 116. The messaging server application 114 implements anumber of message processing technologies and functions, particularlyrelated to the aggregation and other processing of content (e.g.,textual and multimedia content) included in messages received frommultiple instances of the messaging client application 104. As will bedescribed in further detail, the text and media content from multiplesources may be aggregated into collections of content (e.g., calledstories or galleries). These collections are then made available, by themessaging server application 114, to the messaging client application104. Other processor- and memory-intensive processing of data may alsobe performed server-side by the messaging server application 114, inview of the hardware requirements for such processing.

The social network system 116 supports various social networkingfunctions and services, and makes these functions and services availableto the messaging server application 114. To this end, the social networksystem 116 maintains and accesses an entity graph within the database120. Examples of functions and services supported by the social networksystem 116 include the identification of other users of the messagingsystem 100 with whom a particular user has relationships or whom theuser is “following,” and also the identification of other entities andinterests of a particular user.

FIG. 2 is block diagram illustrating further details regarding themessaging system 100, according to exemplary embodiments. Specifically,the messaging system 100 is shown to comprise the messaging clientapplication 104 and the application server 112, which in turn embody anumber of subsystems, namely an ephemeral timer system 202, a collectionmanagement system 204, an annotation system 206, and an encoder-decodersystem 208.

The ephemeral timer system 202 is responsible for enforcing thetemporary access to content permitted by the messaging clientapplication 104 and the messaging server application 114. To this end,the ephemeral timer system 202 incorporates a number of timers that,based on duration and display parameters associated with a message, orcollection of messages (e.g., a SNAPCHAT story, such as the storycomponent 404 discussed below), selectively display and enable access tomessages and associated content via the messaging client application104. Further details regarding the operation of the ephemeral timersystem 202 are provided below.

The collection management system 204 is responsible for managingcollections of media (e.g., collections of text, image, video, and audiodata). In some examples, a collection of content (e.g., messages,including images, video, text, and audio) may be organized into an“event gallery” or an “event story.” Such a collection may be madeavailable for a specified time period, such as the duration of an eventto which the content relates. For example, content relating to a musicconcert may be made available as a “story” for the duration of thatmusic concert. The collection management system 204 may also beresponsible for publishing an icon that provides notification of theexistence of a particular collection to the user interface of themessaging client application 104.

The annotation system 206 provides various functions that enable a userto annotate or otherwise modify or edit media content associated with amessage. For example, the annotation system 206 provides functionsrelated to the generation and publishing of media overlays for messagesprocessed by the messaging system 100. For example, the annotationsystem 206 operatively supplies a media overlay to the messaging clientapplication 104 based on a geolocation of the client device 102. Inanother example, the annotation system 206 operatively supplies a mediaoverlay to the messaging client application 104 based on otherinformation, such as social network information of the user of theclient device 102. A media overlay may include audio and visual contentand visual effects. Examples of audio and visual content includepictures, texts, logos, animations, and sound effects. An example of avisual effect includes color overlaying. The audio and visual content orthe visual effects can be applied to a media content item (e.g., aphoto) at the client device 102. For example, the media overlay mayinclude text that can be overlaid on top of a photograph generated bythe client device 102. In another example, the media overlay includes anidentification of a location (e.g., Venice Beach), a name of a liveevent, or a name of a merchant (e.g., Beach Coffee House). In anotherexample, the annotation system 206 uses the geolocation of the clientdevice 102 to identify a media overlay that includes the name of amerchant at the geolocation of the client device 102. The media overlaymay include other indicia associated with the merchant. The mediaoverlays may be stored in the database 120 and accessed through thedatabase server 118.

In one exemplary embodiment, the annotation system 206 provides auser-based publication platform that enables users to select ageolocation on a map, and upload content associated with the selectedgeolocation. The user may also specify circumstances under which aparticular media overlay should be offered to other users. Theannotation system 206 generates a media overlay that includes theuploaded content and associates the uploaded content with the selectedgeolocation.

In another exemplary embodiment, the annotation system 206 provides amerchant-based publication platform that enables merchants to select aparticular media overlay associated with a geolocation via a biddingprocess. For example, the annotation system 206 associates the mediaoverlay of a highest-bidding merchant with a corresponding geolocationfor a predefined amount of time.

The encoder-decoder system 208 may provide for encoding and decoding ofcustom barcode information. In some aspects, a user of the socialmessaging system 100 may encode an identifier within an image. The imagemay then be generated by the social networking system 100. The disclosedimplementations may also decode the image to extract the encodedidentifier. In some aspects, the identifier may be encoded within aghost image.

FIG. 3 is a schematic diagram 300 illustrating data which may be storedin the database 120 of the messaging server system 108, according tocertain example embodiments. While the content of the database 120 isshown to comprise a number of tables, it will be appreciated that thedata could be stored in other types of data structures (e.g., as anobject-oriented database).

The database 120 includes message data stored within a message table614. An entity table 302 stores entity data, including an entity graph304. Entities for which records are maintained within the entity table302 may include individuals, corporate entities, organizations, objects,places, events, etc. Regardless of type, any entity regarding which themessaging server system 108 stores data may be a recognized entity. Eachentity is provided with a unique identifier, as well as an entity typeidentifier (not shown).

The entity graph 304 furthermore stores information regardingrelationships and associations between or among entities. Suchrelationships may be social, professional (e.g., work at a commoncorporation or organization), interested-based, or activity-based,merely for example.

The database 120 also stores annotation data, in the example form offilters, in an annotation table 312. Filters for which data is storedwithin the annotation table 312 are associated with and applied tovideos (for which data is stored in a video table 310) and/or images(for which data is stored in an image table 308). Filters, in oneexample, are overlays that are displayed as overlaid on an image orvideo during presentation to a recipient user. Filters may be of variestypes, including user-selected filters from a gallery of filterspresented to a sending user by the messaging client application 104 whenthe sending user is composing a message. Other types of filters includegeolocation filters (also known as geo-filters), which may be presentedto a sending user based on geographic location. For example, geolocationfilters specific to a neighborhood or special location may be presentedwithin a user interface by the messaging client application 104, basedon geolocation information determined by a Global Positioning System(GPS) unit of the client device 102. Another type of filter is a datafilter, which may be selectively presented to a sending user by themessaging client application 104, based on other inputs or informationgathered by the client device 102 during the message creation process.Examples of data filters include a current temperature at a specificlocation, a current speed at which a sending user is traveling, abattery life for a client device 102, or the current time.

Other annotation data that may be stored within the image table 308 isso-called “lens” data. A “lens” may be a real-time special effect andsound that may be added to an image or a video.

As mentioned above, the video table 310 stores video data which, in oneembodiment, is associated with messages for which records are maintainedwithin the message table 314. Similarly, the image table 308 storesimage data associated with messages for which message data is stored inthe entity table 302. The entity table 302 may associate variousannotations from the annotation table 312 with various images and videosstored in the image table 308 and the video table 310.

A story table 306 stores data regarding collections of messages andassociated image, video, or audio data, which are compiled into acollection (e.g., a SNAPCHAT story or a gallery). The creation of aparticular collection may be initiated by a particular user (e.g., auser for whom a record is maintained in the entity table 302). A usermay create a “personal story” in the form of a collection of contentthat has been created and sent/broadcast by that user. To this end, theuser interface of the messaging client application 104 may include anicon that is user-selectable to enable a sending user to add specificcontent to his or her personal story.

A collection may also constitute a “live story,” which is a collectionof content from multiple users that is created manually, automatically,or using a combination of manual and automatic techniques. For example,a “live story” may constitute a curated stream of user-submitted contentfrom various locations and events. Users whose client devices havelocation services enabled and who are at a common location or event at aparticular time may, for example, be presented with an option, via auser interface of the messaging client application 104, to contributecontent to a particular live story. The live story may be identified tothe user by the messaging client application 104, based on his or herlocation. The end result is a “live story” told from a communityperspective.

A further type of content collection is known as a “location story,”which enables a user whose client device 102 is located within aspecific geographic location (e.g., on a college or university campus)to contribute to a particular collection. In some embodiments, acontribution to a location story may require a second degree ofauthentication to verify that the end user belongs to a specificorganization or other entity (e.g., is a student on the universitycampus).

The database 310 may also include an operations table 314. Theoperations table 314 may provide a mapping of bit strings to operationsof the social network system 116. For example, the operations mayinclude, for example, adding a friend to an account of a user,displaying a user interface dialog, enabling access to a defined set ofcontent within an account of a user, or other operations. Each of theoperations may be mapped to a unique identifier, such as a bit string.In some of the embodiments disclosed below, a user may create a mappingof a bit string to a particular operation for the user's account. Thebit string may then be encoded into an image, such as a custom graphicas described in more detail below. When the bit string is decoded fromthe custom graphic by a second user, the decoded bit string may serve asan instructions for the social network system 116 to perform theoperation mapped to the bit string via the operations table 315. In someother aspects, the operations table may map a bit string to a series ofinstructions for the social network system 116. For example, theoperations table may map a bit string to a plurality of java byte codes,common intermediate language (CIL) instructions, or SIL. Alternatively,the operations table 314 may map a bit string into scripting languageinstructions, such as any of Perl, python, ruby, java script, or ajaxstatements.

FIG. 4 is a block diagram illustrating functional components of theencoder-decoder system 208 that forms part of the messaging system 100,according to some example embodiments. To avoid obscuring the inventivesubject matter with unnecessary detail, various functional components(e.g., modules, engines, and databases) that are not germane toconveying an understanding of the inventive subject matter have beenomitted from FIG. 4 . However, a skilled artisan will readily recognizethat various additional functional components may be supported by theencoder-decoder system 208 to facilitate additional functionality thatis not specifically described herein. As shown, the encoder-decodersystem 208 includes an encoder 402, decoder 404, loss estimator 406, astyle transfer component 408, and an interpreter 410.

The encoder 402 may receive information indicating a bit string to beencoded within an image. The encoder may then convert the bit stringinformation into visual information encoding the barcode.

The decoder 404 may receive an image included an encoded bit string. Theencoded bit string may be within a graphic in some aspects. The decoder404 may process image data within the graphic and identify an encodedbit string within the image data.

The loss estimator 406 estimates a loss between an encoded version of abit stream and a decoded version of the encoded bit stream. In someaspects, the encoder and/or decoder may be trained based on one or moreloss values determined by the loss estimator 406, as discussed furtherbelow.

The style transfer component 408 may transfer a style component of animage to separate image data encoding a bit string. The image data maybe generated by the encoder 402.

The interpreter 409 may be configured to execute instructions decoded bythe decoder 404. For example, the bit string decoded by the decoder 404may represent intermediate language instructions, such as Java bytecodes, common intermediate language (CIL), or Swift intermediatelanguage (SIL). The decoder 404 may, in some aspects, pass the decodedbit string to the interpreter 409 which may then execute theinstructions included in the bit string. In some other aspects, thedecoded bit string may identify operations to be performed via theoperations table 314. In this case, the interpreter 409 may lookup whichoperations are to be performed by using the operations table 314, andthen cause those operations to be performed.

The above referenced functional components of the encoder-decoder system208 are configured to communicate with each other (e.g., via a bus,shared memory, a switch, or APIs). Collectively, these componentsfacilitate selective modification of content, such as stickers, toinclude a cloud texture, based at least in part, on a location of thecontent within an image.

As is understood by skilled artisans in the relevant computer andInternet-related arts, each functional component illustrated in FIG. 4may be implemented using hardware (e.g., a processor of a machine) or acombination of logic (e.g., executable software instructions) andhardware (e.g., memory and the processor of a machine) for executing thelogic. For example, any component included as part of theencoder-decoder system 208 may physically include an arrangement of oneor more processors 410 (e.g., a subset of or among one or moreprocessors of a machine) configured to perform the operations describedherein for that component. As another example, any component of theencoder-decoder system 208 may include software, hardware, or both, thatconfigure an arrangement of the one or more processors 410 to performthe operations described herein for that component. Accordingly,different components of the encoder-decoder system 208 may include andconfigure different arrangements of such processors 410 or a singlearrangement of such processors 410 at different points in time.

Furthermore, the various functional components depicted in FIG. 4 mayreside on a single machine (e.g., a client device or a server) or may bedistributed across several machines in various arrangements such ascloud-based architectures. Moreover, any two or more of these componentsmay be combined into a single component, and the functions describedherein for a single component may be subdivided among multiplecomponents. Functional details of these components are described belowwith respect to at least FIGS. 5-10 .

FIG. 5 is an example of image data encoding a bit string according to atleast some of the disclosed embodiments. The image data 500 includes avisually appealing graphic 502. In the illustrated embodiment, thevisually appealing graphic is a ghost outline. The graphic 502 includesa border 503 and an interior portion 504. The interior portion 504 maybe substantially or completely surrounded by the border 503. Image datawithin the interior portion 504 may encode a bit string using themethods described in this disclosure. To detect the bit string, someembodiments may first detect the graphic 502 and/or the border 503. Anexample embodiment used to generate image data 500 is discussed in moredetail with respect to FIG. 6 .

FIG. 6 is a flowchart of a method for training an encoder and a decoder.In some aspects, one or more of the functions discussed below withrespect to FIG. 6 may be performed by the processor(s) 410. In someaspects, these processor(s) 410 may be equivalent to the processing unit954 discussed below with respect to FIG. 9 and/or the processor(s) 1004discussed below with respect to FIG. 10 . In some aspects, instructions,such as instructions 904 and/or 1010, discussed below, may configure theprocessor(s) to perform one or more of the functions discussed belowwith respect to process 600.

In block 605, a training bit string is obtained. In some aspects, thetraining bit string may be any bit string.

In block 610, the training bit string is encoded by an encoder. In someaspects, this includes converting an input bit sequence represented bythe bit string into a visual representation of the bit string. Theconversion may be based on a set of encoder parameters. In some aspects,the encoding of the bit string may utilize a multiplicative andup-convolutional layers. In some aspects, the encoder may be based onelement-wise non-linearities. Example encoders may be found in A.Dosovitskuy, J. T. Springenberg, and T. Brox, Learning to generatechairs with convolutional neural networks, Conf. on Computer Vision andPattern Recognition (CVPR), 2015, and M. D. Zeiler, G. W. Taylor, and R.Fergus, Adaptive deconvolutional networks for mid and high level featurelearning, Int. Conf on Computer Vision (ICCV), pp 2018-2025, 2011.

In some aspects, the encoder used in block 610 may be the encoder 402.In some aspects, the encoder may include one or more convolution,polling, and deconvolution layers. Weights may be implemented so as todetermine the convolution and deconvolution layers in a neural networkof the encoder. In some aspects, one or more of the weights may befloating numbers. The weights may be initialized to random numbers, andthen updated in successive iterations of process 600 based on a lossfunction as described below. In some implementations, the encoder mayinclude the following layers described using TensorFlow™ syntax: denselayer at (32*32) units (e.g. a fully connected layer of output 32×32),reshare to 1, 32, 32, 1, convolution of 64 units, 3×3, max pooling at2×2, stride 2, deconvolution at 64, 3×3, stride 2, convolution of 64units, 3×3, max pooling at 2×2 stride 2, deconvolution at 64, 3×3,stride 2, convolution of 64 units, 3×3, max pooling at 2×2 stride 2,deconvolution at 3, 3×3, stride 2. In some aspects, the encoder returnsan image tensor having dimensions of (1,32, 32, 3). In some aspects, ahyperbolic tangent*125+255/2 is returned.

In some aspects, the encoder may encode a bit sequence containing n bitsinto an image having a size of m by m (in other words, an image havingequivalent dimensions on all sides (a square).

In block 615, a style transfer is applied to the encoded bit string. Forexample, a style from a style image may be applied to the encoded bitstring. In some aspects, block 615 may apply the method described inJohnson, Justin, et. al, Perceptual Losses for Real-Time Style Transferand Super-Resolution, Stanford University. For example, block 615 mayrely on two components: an image transformation network f_(W) and a lossnetwork ϕ that is used to define several loss functions l₁, . . . l_(k).The image transformation network may be a deep residual convolutionalneural network parameterized by weights W; it transforms input images xinto output images ŷ via the mapping ŷ=f_(W)(x). Each loss functioncomputes a scalar value l_(i)(ŷ, y_(i)) measuring the difference betweenthe output image ŷ and a target image y_(i). The image transformationnetwork may be trained using a stochastic gradient descent to minimize aweighted combination of loss functions:

$W^{*} = {\arg\min\limits_{w}{E_{x,{\{{yi}\}}}\lbrack {\sum\limits_{i = 1}{\lambda_{i}{l_{i}( {{f_{W}(x)},y_{i}} )}}} \rbrack}}$

Where:

-   -   λ_(i) is a predefined weight for layer i    -   l_(i) is a loss for layer i    -   E is an expectation (e.g. average loss over training images)

To address shortcomings of per-pixel losses and allow loss functions tomeasure perceptual and semantic differences between images, someimplementations apply convolutional neural networks pretrained for imageclassification that have already learned to encode the perceptual andsemantic information to be measured in one or more loss functions.Therefore in some aspects, a network ϕ defines a feature reconstructionloss l_(feat) ^(ϕ) and style reconstruction loss l_(style) ^(ϕ) thatmeasure differences in content and style between images. For each inputimage x we have a content target y_(c) and a style target y_(s). Forstyle transfer the content target y_(c) is the input image x and theoutput image combines the content of x=y_(c) with the style of y_(s); wetrain one network per style target. For super-resolution the input x maybe a low-resolution input, the content target y_(c) may be aground-truth high-resolution image, and style reconstruction loss maynot be used; in some aspects, one network per super-resolution factor istrained.

In some aspects, functions defining the style of the image may be takenfrom Gatys, et. al, Image Style Transfer Using Convolutional NeuralNetworks, available athttps://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf(last visited Jun. 27, 2018), the content of which is herebyincorporated by reference in its entirety. For example, transferring astyle from one image to another may be a problem of texture transfer,and thus, a goal may be to synthesize a texture from a source imagewhile constraining the texture synthesis to preserve semantic content ofa target image. In some aspects, a style of an input image may berepresented via different subsets of a convolutional neutral network.For example, as defined by Gatys, one style representation includes(“conv ‘conv1 1’ (a), ‘conv1 1’ and ‘conv2 1’ (b), ‘conv1 1’, ‘conv2 1’and ‘conv3 1’ (c), ‘conv1 1’, ‘conv2 1’, ‘conv3 1’ and ‘conv4 1’ (d),‘conv1 1’, ‘conv2 1’, ‘conv3 1’, ‘conv4 1’and ‘conv5 1’ (e)). A stylerepresentation of a given image may be designed to match the style ofthe given image on an increasing scale while discarding information ofthe global arrangement of a scene represented by the given image. Somerepresentations of a style of an image may include a parametric texturemodel based on a convolutional neural network and a method that invertsan image representation.

As described by Gaty's, some embodiments may obtain a representation ofa style of an image by utilizing a feature space designed to capturetexture information. This feature space can be built on top of filterresponses in any layer of the network. It may include correlationsbetween different filter responses, where the expectation is taken overthe spatial extent of the feature maps. These feature correlations aregiven in some aspects by a Gram matrix G¹ ∈ R^(Nt×Nl), where G_(ij) ^(l)is the inner product between a vectorized feature maps i and j in layerl:G_(ij) ^(l)Σ_(k)F_(ik) ^(l)F_(jk) ^(l)

As explained by Gatys, by including feature correlations of multiplelayers, stationary multi-scale representation of the input image isobtained, which captures its texture information but not the globalarrangement.

In block 620, the encoded bit string (with the new style) is decoded bya decoder. In some aspects, the encoded bit string is decoded by thedecoder 404. In some aspects, the decoding of block 620 is performed bya recognizer network that is a feedforward convolutional network havingone output unit and n dimensions. The decoder may also have weights thatdefine convolution and deconvolution layers in a neural networkimplemented by the decoder.

The loss function may a vector of differences between each bit in thestring. Such a recognizer network is disclosed in Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R. E. Howward, W. Hubbard, and L. D. Jackel,Backpropagation applied to handwritten zip code recognition, Neuralcomputation, 1(4):541-555, 1989. In some aspects, the recognizer networkincludes the following layers, convolution of 64 filters at 3×3,convolution of 64 filters at 3×3, pooling filter stride 2, 2×2,convolution of 64 filters at 3×3, convolution of 64 filters at 3×3,pooling filter stride 2, 2×2, convolution of 64 filters at 3×3,convolution of 64 filters, 3×3, pooling filter stride 2, 2×2, denselayer at 128 units, dense layer at n units (e.g. n=9). In some aspects,the decoder is configured to return a bit string of n dimensions.

In block 625, an error in the encoding of block 610, style transfer ofblock 615, and decoding of block 620 is measured. In some aspects, theerror is measured by:

${L( {b,r} )} = {{- \frac{1}{n}{\sum\limits_{i = 1}^{n}{\sigma( {b_{i}r_{i}} )}}} = {- \frac{1}{n}{\sum\limits_{i = 1}^{n}\frac{1}{1 + {\exp( {- b_{i}r_{i}} )}}}}}$

where the decoder infers a real-valued sequence r={r₁, r₂, . . . ,r_(n)} from an image containing a representation of the bit stringb={b₁, b₂, . . . b_(n)}, where the bit string b may be encoded by theencoder discussed above with respect to block 610. n is the number ofbits in the bit string. The loss L(b,r) is distributed between −1(perfect recognition) and 0. Another example loss function is providedin Grinchuk, Oleg, et. al, Learnable Visual Markers, Sholkovo Instituteof Science and Technology, Moscow, Russia, and Yandex, Moscow, Russia.

In block 630, an encoder and a decoder are updated to minimize theerror. Minimizing the error may include updating the weights of theencoder and or the decoder discussed above based on the loss. Forexample, in some aspects, block 630 may determine which weights of theencoder and decoder contributed to the error or loss. In some aspects,this may be determined based on a derivative of the loss value for aparticular layer divided by a derivative of a weight for that layer, or,in other words

$\frac{dL}{dW},$where w is a weight at a particular layer and L is a loss or error atthe particular layer. In some aspects, minimizing the loss may includeupdating the weights so that they change in an opposite direction of thegradient. For example, the new weight for a particular layer may bedetermined via:

$W = {w_{i} - {\eta\frac{dL}{dW}}}$

Where:

-   -   W is a weight for a particular layer    -   w_(i) is an initial weight for the layer    -   η is a learning rate parameter (larger rates take larger steps,        and it thus may require fewer iterations to converge on a        trained model. The tradeoff is less a less accurate model when        compared to a lower value of η.

The updated encoder may be the encoder 402. The updated decoder may bethe decoder 404. In some aspects, the minimization of the loss can beaccomplished using a stochastic gradient descent algorithm similar tothe one discussed above with respect to the style transfer. Such analgorithm is disclosed in D. P. Kingma and M. Welling, Auto-encodingvariational bayes, International Conference on Learning Representations,2015. In some aspects, the error is back-propagated to two neuralnetworks (one for encoder and one for decoder), which may update theweights in each of the two neural networks. A normal optimization may beperformed to minimize the loss function indicated by the error.

After block 625, process 600 may return to block 605, which obtainsanother training bit string. Alternatively, after a number ofiterations, process 600 may exit. In some aspects, process 600 may exitwhen iterative losses are no longer being reduced. For example, when adifference between a first loss of a first iteration and a second lossof a subsequent second iteration is below a threshold value, thetraining of the encoder and decoder may be considered complete.

FIG. 7 is a flowchart of a method of encoding a bit string. In someaspects, one or more of the functions discussed below with respect toprocess 700 and FIG. 7 may be performed by the processor(s) 410. In someaspects, these processor(s) 410 may be equivalent to the processing unit954 discussed below with respect to FIG. 9 and/or the processor(s) 1004discussed below with respect to FIG. 10 . In some aspects, instructions,such as instructions 904 and/or 1010, discussed below, may configure theprocessor(s) to perform one or more of the functions discussed belowwith respect to process 700 and FIG. 7 .

In block 705, instructions for a social networking function are encodedinto a bit string. In some aspects, block 705 may be performed on behalfof a first user of the social network system 116. In some aspects, thebit string may literally represent instructions for the social networksystem 116. For example, in some aspects, the bit string may representintermediate code, such as Java byte codes, Common Intermediate Language(CIL) or Swift intermedia language (SIL). In some aspects, the bitstring encodes an identifier, and the identifier maps to one or moresocial networking functions. For example, the social network system 116may maintain a database of identifiers, each of which is mapped to oneor more operations within the social network system 116. One example ofsuch a database is the operations table 314, discussed above withrespect to FIG. 3 . These operations could include, for example, addinga second user that scans the bit string as a friend of the first user ofthe social network system 116, display a particular user interface ofthe social network system 116, which may present one or more functionaloptions to the second user scanning the bit string. For example, a oneuser interface that may be displayed may provide for payment by thescanning second user to a payment account for the first user. Anotheruser interface may provide for displaying particular content within theaccount of the first user. For example, the first user may encode thebit string to identify a particular story or group of content of thefirst user. Upon scanning an image encoding the bit string, the seconduser scanning the bit string may be provided with a user interface thatallows the second user to view the particular story or group of contentidentified by the first user. In some aspects, the bit string is ninebits long. Alternatively, the bit string may store the nine bits in anon-binary format, and thus may be longer than nine bits. In someaspects, the bit string may be any length, and is not limited to nine(9).

In block 710, image data is generated. The image data is generated so asto encode the bit string. The bit string may be encoded by a trainedencoder, such as the encoder 402.

In block 712, a style is transferred to the image data. In some aspects,a sample image including a desired style may be used as a basic for thestyle transfer. As discussed above, the style transfer may beaccomplished by minimizing a difference between a style component of thesample image and a style component of the image data by modifying theimage data. Concurrently, a difference between a content component ofthe modified image data may be minimized with respect to a secondcontent component of the unmodified image data. Some implementations mayapply one or more of the style transfer methods discussed in Johnson,supra. In some aspects, the style component of both the sample image andimage data may be based on feature correlations between multiple layersof a convolutional neural network applied to both the sample image andthe image data. In some aspects, the methods of Gatys, discussed above,may be used to transfer the style of the sample image to the image datawhile preserving content elements of the image data. The content may bepreserved by minimizing a difference between a content function of theimage data below and after modification to incorporate the style of thesample image.

In block 715, a predetermined graphic image is generated. For example,in some aspects, a graphic image such as the graphic 502 shown in FIG. 5may be generated. The graphic may include a border, which substantiallysurrounds a perimeter of the graphic. In some aspects, the border maycompletely surround the perimeter of the graphic.

In block 720, the graphic is filled based on the image data. Forexample, as discussed above with respect to FIG. 5 , the graphic 502 maybe generated, and the graphic 502 may be filled in with image data 504.Filling the graphic based on the image data may include filling aninterior portion of the graphic, for example, within the borderdiscussed above with respect to 715, with the image data. In someaspects, the image data may be repeated as necessary to completely fillthe area within the border. In some aspects, the filled portion of thegraphic may be in black and white form. In other aspects, the graphicmay include color data.

FIG. 8 is a flowchart of a method of decoding a bit string from imagedata. In some aspects, one or more of the functions discussed below withrespect to FIG. 8 may be performed by the processor(s) 410. In someaspects, these processor(s) 410 may be equivalent to the processing unit954 discussed below with respect to FIG. 9 and/or the processor(s) 1004discussed below with respect to FIG. 10 . In some aspects, instructions,such as instructions 904 and/or 1010, discussed below, may configure theprocessor(s) to perform one or more of the functions discussed belowwith respect to process 800 and FIG. 8 .

In block 802, an image is received. The image may be received orcaptured by a device of a first user of the social network system 116.In some aspects, the image may be received from an image sensor, astable storage, or a network. In some aspects, the image received inblock 802 may represent a graphic as generated in block 720. The imagereceived in block 802 may have been created or otherwise encoded by asecond user of the social network system 116. In block 805, a graphic isdetected in an image. For example, as discussed above with respect toFIG. 5 , a graphic (e.g. 502) may be filled with image data (e.g. 504)encoding a bit string in some aspects of this disclosure.

In some aspects, block 805 may search the image data of the image for acustom graphic. In an example embodiment, block 805 searches for thecustom graphic by extracting a candidate shape feature, or multiplecandidate shape features, from the image data. For example, block 805may perform an edge detection technique, or another image processingtechnique, to identify the candidate shape feature such as a contourline of the image. Block 805 then determines whether the candidate shapefeature satisfies shape feature rules or criteria. For instance, if aparticular candidate shape feature is a contour line, the custom patternsystem can determine whether the contour line is an enclosed line thatencircles a portion of the image. Consistent with some embodiments, theshape feature rules filter out irrelevant or extraneous candidate shapefeatures or candidate shape features with a low probability of being thecustom graphic. Alternatively, instead of using a candidate shape torecognize the custom pattern, facial recognition technology can be usedto identify a face.

In response to a candidate shape feature satisfying shape feature rules,block 805 identifies the custom graphic by comparing the candidate shapefeature with a reference shape feature of the custom graphic. Forexample, the custom pattern system can compare an area or size of thecandidate shape feature with a reference area or size of the referenceshape feature. In this example, block 805 identifies the custom graphicbased on a match or near match (e.g., a percentage match above athreshold) between the candidate shape feature and the reference shapefeature.

In block 810 image data within the graphic is detected. For example, theimage data 504 of FIG. 5 may be detected within the graphic 502. In someaspects, the image data may be data inside a custom graphic, such as theimage data 504 inside the custom graphic 502 shown in FIG. 5 . In someaspects, the image data may be detected by first detecting a border thatsubstantially surrounds the custom graphic, with the image datadetermined to be any data within the border.

In block 815, the image data is analyzed to detect a bit string. In someaspects, the decoder 404 may analyze the image data to detect the bitstring. As discussed above, the decoder may be a trained decoder. Thetrained decoder may be a feedforward convolutional network with n outputunits. Such a decoder is described in Y. LeCun, supra. In some aspects,the decoder may extract content data from the image while ignoring styledata. The content data may then be analyzed to detect the bit stringusing the decoder. In some aspects, content may be extracted from theimage data using one or more of the methods of Gatys, discussed above.

Once the bit string is decoded, it may be used to retrieve a variety ofinformation from a data store. For example, the bit string may identifya particular operation to perform for a particular user account in someaspects. For example, in some aspects, the decoded bit string mayidentify one or more operations for the social network system 116 toperform via the operations table 314 in some aspects. In some aspects,the bit string may function to instruct the social network system 116 toadd a first user to an account of a second user. Alternatively, the bitstring may instruct the social network system 116 to display aparticular user interface upon decoding of a custom graphic (e.g. 502)encoding the bit string. For example, the particular user interfaceidentified may display and/or provide a number of options for selectionby a user capturing an image of the custom graphic detected in block805. In some aspects, the bit string may selectively enable or disablecertain features of the social network system 116 with respect to a usercapturing or receiving the image in block 802. For example, the bitstring may instruct the social network system 116 to enable one or moreimage filters for the user receiving or capturing the image of block802. In some aspects, the decoded bit string may be passed to theinterpreter 409, which may execute instructions represented by the bitstring. The instructions may be encoded in the bit string itself, oralternatively, the operations table 314 may store a mapping of the bitstring to a set of instructions to be executed upon decoding of the bitstring. The instructions may be in a variety of forms, including, asdiscussed above, an intermediate language such as Java byte codes, CIL,or SIL. Alternatively, the instructions may be in a source languageformat or script language format such as Perl, Ajax, Ruby, or Pythonsource code statements in some aspects. The instructions may perform anynumber of social networking functions provided by the interpreter 409.For example, the intermediate language encoded in the bit string maycause the social network to display a particular user interface, add afriend to a user scanning the bit string, display a promotion, display aphoto album, or any other social network action.

Software Architecture

FIG. 9 is a block diagram illustrating an example software architecture906, which may be used in conjunction with various hardwarearchitectures herein described. FIG. 9 is a non-limiting example of asoftware architecture and it will be appreciated that many otherarchitectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 906 may execute on hardwaresuch as a machine 1000 of FIG. 10 that includes, among other things,processors 904, memory/storage 906, and I/O components 918. Arepresentative hardware layer 952 is illustrated and can represent, forexample, the machine 1000 of FIG. 10 . The representative hardware layer952 includes a processing unit 954 having associated executableinstructions 904. The executable instructions 904 represent theexecutable instructions of the software architecture 906, includingimplementation of the methods, components, and so forth describedherein. The hardware layer 952 also includes memory and/or storage 956,which also have the executable instructions 904. The hardware layer 952may also comprise other hardware 1258.

As used herein, the term “component” may refer to a device, a physicalentity, or logic having boundaries defined by function or subroutinecalls, branch points, APIs, and/or other technologies that provide forthe partitioning or modularization of particular processing or controlfunctions. Components may be combined via their interfaces with othercomponents to carry out a machine process. A component may be a packagedfunctional hardware unit designed for use with other components and apart of a program that usually performs a particular function of relatedfunctions.

Components may constitute either software components (e.g., codeembodied on a machine-readable medium) or hardware components. A“hardware component” is a tangible unit capable of performing certainoperations and may be configured or arranged in a certain physicalmanner. In various exemplary embodiments, one or more computer systems(e.g., a standalone computer system, a client computer system, or aserver computer system) or one or more hardware components of a computersystem (e.g., a processor or a group of processors) may be configured bysoftware (e.g., an application or application portion) as a hardwarecomponent that operates to perform certain operations as describedherein. A hardware component may also be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware component may include dedicated circuitry or logic that ispermanently configured to perform certain operations.

A hardware component may be a special-purpose processor, such as aField-Programmable Gate Array (FPGA) or an Application-SpecificIntegrated Circuit (ASIC). A hardware component may also includeprogrammable logic or circuitry that is temporarily configured bysoftware to perform certain operations. For example, a hardwarecomponent may include software executed by a general-purpose processoror other programmable processor. Once configured by such software,hardware components become specific machines (or specific components ofa machine) uniquely tailored to perform the configured functions and areno longer general-purpose processors. It will be appreciated that thedecision to implement a hardware component mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

A processor may be, or include, any circuit or virtual circuit (aphysical circuit emulated by logic executing on an actual processor)that manipulates data values according to control signals (e.g.,“commands,” “op codes,” “machine code,” etc.) and that producescorresponding output signals that are applied to operate a machine. Aprocessor may, for example, be a Central Processing Unit (CPU), aReduced Instruction Set Computing (RISC) processor, a ComplexInstruction Set Computing (CISC) processor, a Graphics Processing Unit(GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-FrequencyIntegrated Circuit (RFIC), or any combination thereof. A processor mayfurther be a multi-core processor having two or more independentprocessors (sometimes referred to as “cores”) that may executeinstructions contemporaneously.

Accordingly, the phrase “hardware component” (or “hardware-implementedcomponent”) should be understood to encompass a tangible entity, be thatan entity that is physically constructed, permanently configured (e.g.,hardwired), or temporarily configured (e.g., programmed) to operate in acertain manner or to perform certain operations described herein.Considering embodiments in which hardware components are temporarilyconfigured (e.g., programmed), each of the hardware components need notbe configured or instantiated at any one instance in time. For example,where a hardware component comprises a general-purpose processorconfigured by software to become a special-purpose processor, thegeneral-purpose processor may be configured as respectively differentspecial-purpose processors (e.g., comprising different hardwarecomponents) at different times. Software accordingly configures aparticular processor or processors, for example, to constitute aparticular hardware component at one instance of time and to constitutea different hardware component at a different instance of time. Hardwarecomponents can provide information to, and receive information from,other hardware components. Accordingly, the described hardwarecomponents may be regarded as being communicatively coupled. Wheremultiple hardware components exist contemporaneously, communications maybe achieved through signal transmission (e.g., over appropriate circuitsand buses) between or among two or more of the hardware components. Inembodiments in which multiple hardware components are configured orinstantiated at different times, communications between or among suchhardware components may be achieved, for example, through the storageand retrieval of information in memory structures to which the multiplehardware components have access.

For example, one hardware component may perform an operation and storethe output of that operation in a memory device to which it iscommunicatively coupled. A further hardware component may then, at alater time, access the memory device to retrieve and process the storedoutput. Hardware components may also initiate communications with inputor output devices, and can operate on a resource (e.g., a collection ofinformation). The various operations of example methods described hereinmay be performed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implementedcomponents that operate to perform one or more operations or functionsdescribed herein. As used herein, “processor-implemented component”refers to a hardware component implemented using one or more processors.Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a particular processor or processors beingan example of hardware. For example, at least some of the operations ofa method may be performed by one or more processors orprocessor-implemented components.

Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines including processors), with these operationsbeing accessible via a network (e.g., the Internet) and via one or moreappropriate interfaces (e.g., an API). The performance of certain of theoperations may be distributed among the processors, not only residingwithin a single machine, but deployed across a number of machines. Insome exemplary embodiments, the processors or processor-implementedcomponents may be located in a single geographic location (e.g., withina home environment, an office environment, or a server farm). In otherexemplary embodiments, the processors or processor-implementedcomponents may be distributed across a number of geographic locations.

In the exemplary architecture of FIG. 9 , the software architecture 906may be conceptualized as a stack of layers where each layer providesparticular functionality. For example, the software architecture 906 mayinclude layers such as an operating system 902, libraries 920,frameworks/middleware 918, applications 916, and a presentation layer914. Operationally, the applications 916 and/or other components withinthe layers may invoke API calls 908 through the software stack andreceive a response as messages 910. The layers illustrated arerepresentative in nature and not all software architectures have alllayers. For example, some mobile or special-purpose operating systemsmay not provide a frameworks/middleware 918 layer, while others mayprovide such a layer. Other software architectures may includeadditional or different layers.

The operating system 902 may manage hardware resources and providecommon services. The operating system 902 may include, for example, akernel 922, services 924, and drivers 926. The kernel 922 may act as anabstraction layer between the hardware and the other software layers.For example, the kernel 922 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 924 may provideother common services for the other software layers. The drivers 926 areresponsible for controlling or interfacing with the underlying hardware.For instance, the drivers 926 include display drivers, camera drivers,Bluetooth® drivers, flash memory drivers, serial communication drivers(e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audiodrivers, power management drivers, and so forth depending on thehardware configuration.

The libraries 920 provide a common infrastructure that is used by theapplications 916 and/or other components and/or layers. The libraries920 provide functionality that allows other software components toperform tasks in an easier fashion than by interfacing directly with theunderlying operating system 902 functionality (e.g., kernel 922,services 924, and/or drivers 926). The libraries 920 may include systemlibraries 944 (e.g., C standard library) that may provide functions suchas memory allocation functions, string manipulation functions,mathematical functions, and the like. In addition, the libraries 920 mayinclude API libraries 946 such as media libraries (e.g., libraries tosupport presentation and manipulation of various media formats such asMPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., anOpenGL framework that may be used to render 2D and 3D graphic content ona display), database libraries (e.g., SQLite that may provide variousrelational database functions), web libraries (e.g., WebKit that mayprovide web browsing functionality), and the like. The libraries 920 mayalso include a wide variety of other libraries 948 to provide many otherAPIs to the applications 916 and other software components/modules.

The frameworks/middleware 918 provide a higher-level commoninfrastructure that may be used by the applications 916 and/or othersoftware components/modules. For example, the frameworks/middleware 918may provide various graphic user interface (GUI) functions, high-levelresource management, high-level location services, and so forth. Theframeworks/middleware 918 may provide a broad spectrum of other APIsthat may be utilized by the applications 916 and/or other softwarecomponents/modules, some of which may be specific to a particularoperating system 902 or platform.

The applications 916 include built-in applications 938 and/orthird-party applications 940. Examples of representative built-inapplications 938 may include, but are not limited to, a contactsapplication, a browser application, a book reader application, alocation application, a media application, a messaging application,and/or a game application. The third-party applications 940 may includean application developed using the ANDROID™ or IOS™ software developmentkit (SDK) by an entity other than the vendor of the particular platform,and may be mobile software running on a mobile operating system such asIOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Thethird-party applications 940 may invoke the API calls 908 provided bythe mobile operating system (such as the operating system 902) tofacilitate functionality described herein.

The applications 916 may use built-in operating system functions (e.g.,kernel 922, services 924, and/or drivers 926), libraries 920, andframeworks/middleware 918 to create user interfaces to interact withusers of the system. Alternatively, or additionally, in some systemsinteractions with a user may occur through a presentation layer, such asthe presentation layer 914. In these systems, the application/component“logic” can be separated from the aspects of the application/componentthat interact with a user.

Exemplary Machine

FIG. 10 is a block diagram illustrating exemplary components (alsoreferred to herein as “modules”) of a machine 1000. In some aspects, themachine is configured to read instructions from a machine-readablemedium (e.g., a machine-readable storage medium) and perform any one ormore of the methodologies discussed herein. Specifically, FIG. 10 showsa diagrammatic representation of the machine 1000 in the example form ofa computer system, within which instructions 1010 (e.g., software, aprogram, an application, an applet, an app, or other executable code)for causing the machine 1000 to perform any one or more of themethodologies discussed herein may be executed. As such, theinstructions 1010 may be used to implement modules or componentsdescribed herein. The instructions 1010 transform the general,non-programmed machine 1000 into a particular machine 1000 programmed tocarry out the described and illustrated functions in the mannerdescribed. In alternative embodiments, the machine 1000 operates as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 1000 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1000 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), apersonal digital assistant (PDA), an entertainment media system, acellular telephone, a smart phone, a mobile device, a wearable device(e.g., a smart watch), a smart home device (e.g., a smart appliance),other smart devices, a web appliance, a network router, a networkswitch, a network bridge, or any machine capable of executing theinstructions 1010, sequentially or otherwise, that specify actions to betaken by machine 1000. Further, while only a single machine 1000 isillustrated, the term “machine” shall also be taken to include acollection of machines that individually or jointly execute theinstructions 1010 to perform any one or more of the methodologiesdiscussed herein.

The machine 1000 may include processors 1004, memory/storage 1006, andI/O components 1018, which may be configured to communicate with eachother such as via a bus 1002. The memory/storage 1006 may include amemory 1014, such as a main memory, or other memory storage, and astorage unit 1016, both accessible to the processors 1004 such as viathe bus 1002. The storage unit 1016 and memory 1014 store theinstructions 1010 embodying any one or more of the methodologies orfunctions described herein. The instructions 1010 may also reside,completely or partially, within the memory 1014, within the storage unit1016, within at least one of the processors 1004 (e.g., within theprocessor's cache memory), or any suitable combination thereof, duringexecution thereof by the machine 1000. Accordingly, the memory 1014, thestorage unit 1016, and the memory of the processors 1004 are examples ofmachine-readable media. In some aspect, the processors 202 andprocessors 1004 may be the same processors.

As used herein, the term “machine-readable medium,” “computer-readablemedium,” or the like may refer to any component, device, or othertangible medium able to store instructions and data temporarily orpermanently. Examples of such media may include, but are not limited to,random-access memory (RAM), read-only memory (ROM), buffer memory, flashmemory, optical media, magnetic media, cache memory, other types ofstorage (e.g., Electrically Erasable Programmable Read-Only Memory(EEPROM)), and/or any suitable combination thereof. The term“machine-readable medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store instructions. The term“machine-readable medium” may also be taken to include any medium, orcombination of multiple media, that is capable of storing instructions(e.g., code) for execution by a machine, such that the instructions,when executed by one or more processors of the machine, cause themachine to perform any one or more of the methodologies describedherein. Accordingly, a “machine-readable medium” may refer to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” excludes transitory signals per se.

The I/O components 1018 may include a wide variety of components toprovide a user interface for receiving input, providing output,producing output, transmitting information, exchanging information,capturing measurements, and so on. The specific I/O components 1018 thatare included in the user interface of a particular machine 1000 willdepend on the type of machine. For example, portable machines such asmobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 1018 may include many other components that are not shown inFIG. 10 . The I/O components 1018 are grouped according to functionalitymerely for simplifying the following discussion and the grouping is inno way limiting. In various exemplary embodiments, the I/O components1018 may include output components 1026 and input components 1028. Theoutput components 1026 may include visual components (e.g., a displaysuch as a plasma display panel (PDP), a light emitting diode (LED)display, a liquid crystal display (LCD), a projector, or a cathode raytube (CRT)), acoustic components (e.g., speakers), haptic components(e.g., a vibratory motor, resistance mechanisms), other signalgenerators, and so forth. The input components 1028 may includealphanumeric input components (e.g., a keyboard, a touch screenconfigured to receive alphanumeric input, a photo-optical keyboard, orother alphanumeric input components), point-based input components(e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, orother pointing instruments), tactile input components (e.g., a physicalbutton, a touch screen that provides location and/or force of touches ortouch gestures, or other tactile input components), audio inputcomponents (e.g., a microphone), and the like. The input components 1028may also include one or more image-capturing devices, such as a digitalcamera for generating digital images and/or video.

In further exemplary embodiments, the I/O components 1018 may includebiometric components 1030, motion components 1034, environmentcomponents 1036, or position components 1038, as well as a wide array ofother components. For example, the biometric components 1030 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion components 1034 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environment components 1036 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatmay provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 1038 mayinclude location sensor components (e.g., a GPS receiver component),altitude sensor components (e.g., altimeters or barometers that detectair pressure from which altitude may be derived), orientation sensorcomponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1018 may include communication components 1040operable to couple the machine 1000 to a network 1032 or devices 1020via a coupling 1024 and a coupling 1022 respectively. For example, thecommunication components 1040 may include a network interface componentor other suitable device to interface with the network 1032. In furtherexamples, the communication components 1040 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 1020 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 1040 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1040 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF4111, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components1040, such as location via Internet Protocol (IP) geo-location, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

Where a phrase similar to “at least one of A, B, or C,” “at least one ofA, B, and C,” “one or more of A, B, or C,” or “one or more of A, B, andC” is used, it is intended that the phrase be interpreted to mean that Aalone may be present in an embodiment, B alone may be present in anembodiment, C alone may be present in an embodiment, or any combinationof the elements A, B, and C may be present in a single embodiment; forexample, A and B, A and C, B and C, or A and B and C may be present.

Changes and modifications may be made to the disclosed embodimentswithout departing from the scope of the present disclosure. These andother changes or modifications are intended to be included within thescope of the present disclosure, as expressed in the following claims.

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

We claim:
 1. A method comprising: accessing a set of bits; generating,using an encoder, image data encoding the set of bits; incorporating, byhardware processing circuitry, characteristics of a style from a sampleimage into the image data, wherein the incorporating the characteristicsof the style from the sample image into the image data comprisesminimizing a loss function of the style by modifying the image data;generating a graphic image based on the image data that encodes the setof bits and comprises characteristics of the style; associating the setof bits with an operation to be performed by an interaction system upondecoding of the encoded set of bits, wherein the set of bits is includedin a bit string, the method further comprising: accessing the bitstring; generating, using an encoder, image data encoding the bitstring, the encoder trained to minimize a loss in relation to trainingdata generated by the encoder and decoded by a decoder; associating thebit string with the operation to be performed by the interaction systemupon decoding of the bit string; transferring the style from the sampleimage to the image data, wherein the transferring of the style from thesample image to the image data comprises minimizing a loss function ofthe style and a second style of the image data by modifying the imagedata, and minimizing a second loss function of a content of the imagedata and a second content of the modified image data; and filling agraphic image with the image data encoding the bit string and having thestyle.
 2. The method of claim 1, wherein the training data comprises aplurality of training bit strings.
 3. The method of claim 1, wherein thedecoder is also trained to minimize loss in relation to the trainingdata being decoded by the decoder.
 4. The method of claim 1, furthercomprising: generating the graphic image; and writing the filled graphicimage to an output device.
 5. The method of claim 1, wherein the styleof the sample image and a second style of the image data are based onfeature correlations between multiple layers of a convolutional neuralnetwork applied to each of the sample image and the image datarespectively.
 6. The method of claim 1, wherein the encoder includesconvolutional layers comprising: a) dense layer at (32*32) units b)reshape to 1, 32, 32, 1 c) convolution of 64 units, 3×3 d) max poolingat 2×2, stride 2 e) deconvolution at 64, 3×3, stride 2 f) convolution of64 units, 3×3 g) max pooling at 2×2 stride 2 h) deconvolution at 64,3×3, stride 2 i) convolution of 64 units, 3×3 j) max pooling at 2×2stride 2 k) deconvolution at 3, 3×3, stride
 2. 7. The method of claim 1,wherein the encoder is configured to return an image tensor of (1, 32,32, 3) based on a hyperbolic tangent.
 8. The method of claim 1, whereinthe decoder includes layers comprising: a) convolution of 64 filters at3×3 b) convolution of 64 filters at 3×3 c) pooling filter stride 2, 2×2d) convolution of 64 filters at 3×3 e) convolution of 64 filters at 3×3f) pooling filter stride 2, 2×2 g) convolution of 64 filters at 3×3 h)convolution of 64 filters, 3×3 i) pooling filter stride 2, 2×2 j) denselayer at 128 units k) dense layer at 9 units.
 9. The method of claim 1,wherein the decoder is configured to return a bit string tensor of shape(1,1,1,9).
 10. The method of claim 1, wherein the loss is minimized viaa stochastic gradient descent algorithm.
 11. The method of claim 10,further comprising encoding instructions for the interaction system intothe bit string.
 12. The method of claim 1, wherein minimizing the losscomprises updating a first set of weights defining convolution anddeconvolution layers of the encoder based on the loss and updating asecond set of weights defining convolution and deconvolution layers ofthe decoder based on the loss.
 13. A system, comprising: hardwareprocessing circuitry; a hardware memory storing instructions that whenexecuted configure the hardware processing circuitry to performoperations comprising: accessing a set of bits; generating, using anencoder, image data encoding the set of bits; incorporating, by hardwareprocessing circuitry, characteristics of a style from a sample imageinto the image data, wherein the incorporating the characteristics ofthe style from the sample image into the image data comprises minimizinga loss function of the style by modifying the image data; generating agraphic image based on the image data that encodes the set of bits andcomprises characteristics of the style; associating the set of bits withan operation to be performed by an interaction system upon decoding ofthe encoded set of bits, wherein the set of bits is included in a bitstring, the operations further comprising: accessing the bit string;generating, using an encoder, image data encoding the bit string, theencoder trained to minimize a loss in relation to training datagenerated by the encoder and decoded by a decoder; associating the bitstring with the operation to be performed by the interaction system upondecoding of the bit string; transferring the style from the sample imageto the image data, wherein the transferring of the style from the sampleimage to the image data comprises minimizing a loss function of thestyle and a second style of the image data by modifying the image data,and minimizing a second loss function of a content of the image data anda second content of the modified image data; and filling a graphic imagewith the image data encoding the bit string and having the style. 14.The system of claim 13, wherein training data comprises a plurality oftraining bit strings.
 15. The system of claim 13, wherein the decoder isalso trained to minimize loss in relation to the training data beingdecoded by the decoder.
 16. A non-transitory computer readable mediumcomprising non-transitory computer readable instructions that, whenexecuted by one or more processors, configured the one or moreprocessors to perform operations comprising: accessing a set of bits;generating, using an encoder, image data encoding the set of bits;incorporating, by hardware processing circuitry, characteristics of astyle from a sample image into the image data, wherein the incorporatingthe characteristics of the style from the sample image into the imagedata comprises minimizing a loss function of the style by modifying theimage data; generating a graphic image based on the image data thatencodes the set of bits and comprises characteristics of the style;associating the set of bits with an operation to be performed by aninteraction system upon decoding of the encoded set of bits, wherein theset of bits is included in a bit string, the operations furthercomprising: accessing the bit string; generating, using an encoder,image data encoding the bit string, the encoder trained to minimize aloss in relation to training data generated by the encoder and decodedby a decoder; associating the bit string with the operation to beperformed by the interaction system upon decoding of the bit string;transferring the style from the sample image to the image data, whereinthe transferring of the style from the sample image to the image datacomprises minimizing a loss function of the style and a second style ofthe image data by modifying the image data, and minimizing a second lossfunction of a content of the image data and a second content of themodified image data; and filling a graphic image with the image dataencoding the bit string and having the style.